{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Engineering mixed variables\n",
    "\n",
    "We've seen that mixed variables are those which values contain both numbers and labels (see section 2: types of variables for a re-cap).\n",
    "\n",
    "How can we engineer this type of variable to use it in machine learning?\n",
    "\n",
    "What we need to do in these cases is extract the categorical part in one variable, and the numerical part in a different variable. Therefore, we obtain 2 variables from the original one.\n",
    "\n",
    "**Scenario 1**\n",
    "Some mixed variables, will contain among their values either strings or numbers, but not the 2 together. This means that the value of an observation is either a string, or a number, but not a string and a number.\n",
    "\n",
    "There are cases, like I explained in section 2, in which the variable is in nature numerical, and whenever the number could not be established for an observation, a code (i.e., a string) is utilised to define that a) information could not be retrieved and b) why it could not be retrieved.\n",
    "\n",
    "In situations like this, we can simply extract the string into a categorical variable, and the numbers into a numerical variable. We will end up with 2 variables that contain all the information contained in the single mixed variable. After this, we should engineer the numerical and categorical variables as we would do normally for those types of variables.\n",
    "\n",
    "**Scenario 2**\n",
    "In other cases, the variable will contain strings and numbers together for the same observation. After inspecting the variable, we can decide whether we should treat these types as categorical variables, or whether it will be more predictive if we separate the categorical and the numerical parts in different variables.\n",
    "\n",
    "\n",
    "We will see below examples of these 2 scenarios, and how we can go about them.\n",
    "\n",
    "For the first scenario I will simulate some data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# let's load the Lending Club dataset with the variable \"Number of installment accounts opened in past 24 months\"\n",
    "# installment accounts are those that, at the moment of acquiring them, there is a set period and amount\n",
    "# of repayments agreed between the lender and borrower. An example of this is a car loan, or a student loan.\n",
    "# the borrower knows that they are going to pay a certain, fixed amount over, for example 36 months.\n",
    "\n",
    "data = pd.read_csv('loan.csv', usecols=['id','open_il_24m'])\n",
    "\n",
    "# let's replace the NaN with the fictitious codes described below:\n",
    "# 'A': couldn't identify the person\n",
    "# 'B': no relevant data\n",
    "# 'C': person seems not to have any account open\n",
    "# this is exactly what we did in section 2 of this course for the lecture on mixed types of variables\n",
    "\n",
    "# select which observations we will replace with each code\n",
    "indeces_b = data[data.open_il_24m.isnull()].sample(100000, random_state=44).index\n",
    "indeces_c = data[data.open_il_24m.isnull()].sample(300000, random_state=42).index\n",
    "\n",
    "# replace NA with the fictitious code\n",
    "data.open_il_24m.fillna('A', inplace=True)\n",
    "data.loc[indeces_b, 'open_il_24m']='B'\n",
    "data.loc[indeces_c, 'open_il_24m']='C'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['C', 'A', 'B', 0.0, 1.0, 2.0, 4.0, 3.0, 6.0, 5.0, 9.0, 7.0, 8.0,\n",
       "       13.0, 10.0, 19.0, 11.0, 12.0, 14.0, 15.0], dtype=object)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's inspect the mixed variable\n",
    "\n",
    "data.open_il_24m.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variable now is mixed: it contains both numbers and the three codes that we just created for this simulation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0xa629a6ee10>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEVCAYAAAA2IkhQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu8VXWd//HXG0HyAgiKpAJiQjVqqcmgpb/SmJCy1Ck1\n7SI6eBud0SabwpoZSqN0Jq1xUkcbyctkSjYpao7itTFHhLwhXkYSFPECAor3BD+/P77fLYvtOWev\nA2eds928n4/Hepy1v2t9v/uz1t5nf/b6ru9eSxGBmZlZlXr1dABmZtb6nGzMzKxyTjZmZlY5Jxsz\nM6uck42ZmVXOycbMzCrnZGNdTtJFkr7fQ88tST+XtFzS3W0s/7KkG3sgrtskHZXnj5B0R3fHYNaT\nnGzWA5IWSFosaZNC2VGSbuvBsKqyF/ApYGhEjKlfGBG/iIhx6/okkkLSyHVtpztJ2lvSUz0dR1fp\nyS811nlONuuPDYCTejqIzpK0QSerbAssiIhXqojHzNaOk83641+Ab0jarH6BpBH5m3rvQll9t8/v\nJf1Y0guSHpf0sVy+MB81TahrdgtJMyS9JOl2SdsW2v5gXrZM0qOSDiksu0jSeZJ+K+kVYJ824t1a\n0vRcf56ko3P5ROA/gI9KelnS99qou0YXVt7u4yQ9lrftHEnKy0bm2F+U9LykK3L573L1+/PzfFHS\nQEnXSlqSu/CulTS08cvydgzH5xheknSapO0l3SlphaRpkjYsrP9ZSffleO+U9OHCsgWSviHpgRz3\nFZLek49qrwe2zjG/LGnrNmLZT9K9+XkXSvpu3fK98nO+kJcfkcs3knSmpCfy894haaO8bH9Jc3Od\n2yT9Wd22jyw8fvtopXYkJunk/B57RtKRedkxwJeBb+ZtuSaXf0vSorwfH5U0tp19PkDSJfn1ekLS\nP0jqlZfV3u8/zdvySLGdXPfCHM8iSd9X/lJUe39J+lF+H8yX9Oky74OWFxGeWnwCFgB/AfwX8P1c\ndhRwW54fAQTQu1DnNuCoPH8EsBI4knSE9H3gSeAcoC8wDngJ2DSvf1F+/PG8/F+BO/KyTYCFua3e\nwK7A88AOhbovAnuSvgy9p43t+R1wLvAeYBdgCfDJQqx3dLAv1liet/taYDNgeG5rfF72S+A7tTiA\nverqjSw83hz4ArAx0A/4FXBVB/uzPoargf7AjsAbwM3A+4ABwEPAhLzursBiYPf8WkzIr2/fwmt9\nN7A1MAh4GDguL9sbeKrBe2Vv4EN5mz8MPAccmJdtm1/Xw4A+eZt3ycvOydu4TY7rY/m1fz/wCqlr\nsw/wTWAesGE7+/EiVr9H9ya9707NdT8DvAoMrF83P/4A6b21deF9vX0723lJ3uf98nr/B0yse7//\nXX7eL5Lek4Py8t8A55Pey1vm/X1soe6bwNF5P/w18DSgnv4c6OnJRzbrl38C/lbS4LWoOz8ifh4R\nq4ArgGHAqRHxRkTcCPwJKJ7DuC4ifhcRb5A+sD8qaRjwWVI3188jYmVE3Av8Gji4UPfqiPh9RLwV\nEa8Xg8ht7Al8KyJej4j7SEczh6/FNtWcHhEvRMSTwK2kBAbpQ2Nb0ofX6xHR7kn9iFgaEb+OiFcj\n4iVgCvCJTsTwzxGxIiLmAg8CN0bE4xHxIumIZNe83jHA+RExMyJWRcTFpOS0R6GtsyPi6YhYBlxT\n2J6GIuK2iJiT9/0DpIRb244vATdFxC8j4s28zfflI4K/Ak6KiEU5rjvza/9F0nthRkS8CfwI2IiU\njMp4k/Q+ezMifgu8TEoqbVlFSnA7SOoTEQsi4o/1K+WjkEOBUyLipYhYAJwJfLWw2mLgJ/l5rwAe\nBfaTNISU9L4WEa9ExGLgx7m9mici4mf5f+ViYCtgSMntbVlONuuRiHiQ9C1+0lpUf64w/1pur75s\n08LjhYXnfRlYRvq2vS2we+5SeUHSC6TukPe2VbcNWwPL8gd6zROkb9Rr69nC/Kus3o5vAgLuzt1A\nf9VeA5I2lnR+7pJZQTr62kzlzznV78v29u22wMl1+28Yab802p6GJO0u6dbcvfQicBywRV48DHjH\nh3de/p52lm1Nen0AiIi3SK9v2ddraUSsLDxud3siYh7wNeC7wGJJl7fVVZjj7VOMi3e+hxZFRNQt\nr71/+wDPFPb/+aQjnJq3939EvJpnS78GrcrJZv0zmXSIX/zHqp1M37hQVvzwXxvDajOSNiV16TxN\n+qC5PSI2K0ybRsRfF+p2dCnyp4FBkvoVyoYDi9Yx3neIiGcj4uiI2Bo4FjhX7Y9AO5n0jXv3iOhP\n6kKElKy60kJgSt3+2zgiflmibplLvF8GTAeGRcQA4N9ZvQ0Lge3bqPM88Ho7y54mfUADaWg66b1R\ne71eZe3fd+/Ynoi4LCL2ys8ZwBntxFs7aq2pfw9tk2MtLq+9f98Atijs//4RsWMn4l4vOdmsZ/K3\nvyuAEwtlS0j/aF+RtEH+Bt/WB0dnfCafTN4QOA24KyIWko6s3i/pq5L65OnPiyeNG8S/ELgT+GE+\n8f1hYCLwn+sY7ztIOlirT/IvJ314vZUfP0c6p1LTj3QE8oKkQaSkXoWfAcflIxBJ2iSf1O/XsGaK\neXNJAzpYpx/pyPF1SWNIXWc1vwD+QtIhknpL2lzSLvloZSpwltLgjQ0kfVRSX2AaqftprKQ+pKT8\nBuk1BLgP+FKuM57OdT2u8RpI+oCkT+bnfZ30erxVXyl3b00DpkjqpzR45eus+R7aEjgxvz8PBv4M\n+G1EPAPcCJwpqb+kXkqDOToT93rJyWb9dCrp5GbR0cDfA0tJJ6nvrK/USZeRPnCXAbsBXwHI3V/j\nSH3cT5O6HM4g9bWXdRjppO7TpJO1kyPipnWMty1/DsyU9DLp2/5JEfF4XvZd4OLclXII8BPSuYjn\ngbuA/64gHiJiNum1+ikpAc4jnZQuU/cR0jmYx3PcbXUxHQ+cKukl0jm+aYX6T5LOV5xMel3vA3bO\ni78BzAFm5WVnAL0i4lHSa/9vpH3zOeBzEfGnXO+kXFbrTr2qzLZkF5LOz7wg6SrSe+j0/DzPkhLG\nKe3U/VvSEf3jwB2k9+vUwvKZwKjc1hTgoIhYmpcdDmxIGrixHLiSdF7GOqA1uyXNzNZvSsO5j8rd\ncdZFfGRjZmaVc7IxM7PKuRvNzMwq5yMbMzOrnJONmZlVrnfjVdYPW2yxRYwYMaKnwzAze1f5wx/+\n8HxENLwElpNNNmLECGbPnt3TYZiZvatIeqLxWu5GMzOzbuBkY2ZmlXOyMTOzyjnZmJlZ5ZxszMys\ncpUmG6X7oc9Rul/67Fw2SOn+84/lvwML65+idE/5RyXtWyjfLbczT9LZtftMSOqrdI/1eZJmShpR\nqDMhP8djkiZUuZ1mZtax7jiy2ScidomI0fnxJODmiBhFus/6JABJO5AuO78jMJ50o6raXQ7PI11W\nfVSexufyicDyiBhJujXrGbmt2v1EdgfGAJOLSc3MzLpXT3SjHUC6Lzf574GF8svzPe3nk+7TMUbS\nVkD/iLgr36b1kro6tbauBMbmo559gRkRsSwilgMzWJ2gzMysm1X9o84AbpK0Cjg/Ii4AhuS73UG6\nwdGQPL8N6aZTNU/lsjfzfH15rc5CgIhYme+ZvnmxvI06pY2YdF2Hyxecvl9nmzQzWy9VnWz2iohF\nkrYEZkh6pLgwIkJSj112WtIxwDEAw4cP76kwzMxaXqXdaBGxKP9dTLp97xjgudw1Rv67OK++CBhW\nqD40ly3K8/Xla9SR1BsYQLqtcXtt1cd3QUSMjojRgwc3vLSPmZmtpcqSjaRNJPWrzZPuO/8g6V7u\ntdFhE4Cr8/x04NA8wmw70kCAu3OX2wpJe+TzMYfX1am1dRBwSz6vcwMwTtLAPDBgXC4zM7MeUGU3\n2hDgN3mUcm/gsoj4b0mzgGmSJgJPAIcARMRcSdOAh4CVwAkRsSq3dTxwEbARcH2eAC4ELpU0D1hG\nGs1GRCyTdBowK693akQsq3BbzcysA5Ulm4h4HNi5jfKlwNh26kwBprRRPhvYqY3y14GD22lrKjC1\nc1GbmVkVfAUBMzOrnJONmZlVzsnGzMwq52RjZmaVc7IxM7PKOdmYmVnlnGzMzKxyTjZmZlY5Jxsz\nM6uck42ZmVXOycbMzCrnZGNmZpVzsjEzs8o52ZiZWeWcbMzMrHJONmZmVjknGzMzq5yTjZmZVc7J\nxszMKudkY2ZmlXOyMTOzyjnZmJlZ5ZxszMysck42ZmZWOScbMzOrnJONmZlVzsnGzMwq52RjZmaV\nc7IxM7PKOdmYmVnlnGzMzKxyTjZmZla5ypONpA0k3Svp2vx4kKQZkh7LfwcW1j1F0jxJj0rat1C+\nm6Q5ednZkpTL+0q6IpfPlDSiUGdCfo7HJE2oejvNzKx93XFkcxLwcOHxJODmiBgF3JwfI2kH4FBg\nR2A8cK6kDXKd84CjgVF5Gp/LJwLLI2Ik8GPgjNzWIGAysDswBphcTGpmZta9Kk02koYC+wH/USg+\nALg4z18MHFgovzwi3oiI+cA8YIykrYD+EXFXRARwSV2dWltXAmPzUc++wIyIWBYRy4EZrE5QZmbW\nzao+svkJ8E3grULZkIh4Js8/CwzJ89sACwvrPZXLtsnz9eVr1ImIlcCLwOYdtGVmZj2gsmQj6bPA\n4oj4Q3vr5COVqCqGRiQdI2m2pNlLlizpqTDMzFpelUc2ewL7S1oAXA58UtJ/As/lrjHy38V5/UXA\nsEL9oblsUZ6vL1+jjqTewABgaQdtrSEiLoiI0RExevDgwWu/pWZm1qHKkk1EnBIRQyNiBOnE/y0R\n8RVgOlAbHTYBuDrPTwcOzSPMtiMNBLg7d7mtkLRHPh9zeF2dWlsH5ecI4AZgnKSBeWDAuFxmZmY9\noHcPPOfpwDRJE4EngEMAImKupGnAQ8BK4ISIWJXrHA9cBGwEXJ8ngAuBSyXNA5aRkhoRsUzSacCs\nvN6pEbGs6g0zM7O2dUuyiYjbgNvy/FJgbDvrTQGmtFE+G9ipjfLXgYPbaWsqMHVtYzYzs67TsBtN\n0vaS+ub5vSWdKGmz6kMzM7NWUeacza+BVZJGAheQTrxfVmlUZmbWUsokm7fyb1j+Evi3iPh7YKtq\nwzIzs1ZSJtm8Kekw0qiva3NZn+pCMjOzVlMm2RwJfBSYEhHz87DkS6sNy8zMWkmHo9HyhTC/ExFf\nrpXl65adUXVgZmbWOjo8ssm/c9lW0obdFI+ZmbWgMr+zeRz4vaTpwCu1wog4q7KozMyspZRJNn/M\nUy+gX7XhmJlZK2qYbCLiewCSNo6IV6sPyczMWk2ZKwh8VNJDwCP58c6Szq08MjMzaxllhj7/hHTn\ny6UAEXE/8PEqgzIzs9ZS6hYDEbGwrmhVmyuamZm1ocwAgYWSPgaEpD7AScDD1YZlZmatpMyRzXHA\nCcA2pLtd7pIfm5mZlVLmyObl4hUEzMzMOqtMsnlQ0nPA/+Tpjoh4sdqwzMyslTTsRouIkcBhwBxg\nP+B+SfdVHZiZmbWOhkc2koYCewL/D9gZmAvcUXFcZmbWQsp0oz0JzAJ+EBHHVRyPmZm1oDKj0XYF\nLgG+JOl/JV0iaWLFcZmZWQspc220+yXVLsb5/4CvAJ8ALqw4NjMzaxFlztnMBvoCd5JGo308Ip6o\nOjAzM2sdZc7ZfDoillQeiZmZtawy52z+JOksSbPzdKakAZVHZmZmLaNMspkKvAQckqcVwM+rDMrM\nzFpLmW607SPiC4XH3/OPOs3MrDPKHNm8Jmmv2gNJewKvVReSmZm1mjJHNscBlxTO0ywHJlQXkpmZ\ntZoOk42kXsAHImJnSf0BImJFt0RmZmYto8NutIh4C/hmnl/hRGNmZmujzDmbmyR9Q9IwSYNqU+WR\nmZlZyyhzzuaL+W/x7pwBvK/rwzEzs1bU4ZFNPmfzlYjYrm5qmGgkvUfS3ZLulzRX0vdy+SBJMyQ9\nlv8OLNQ5RdI8SY9K2rdQvpukOXnZ2ZKUy/tKuiKXz5Q0olBnQn6OxyR5QIOZWQ8qc87mp2vZ9hvA\nJyNiZ2AXYLykPYBJwM0RMQq4OT9G0g7AocCOwHjgXEkb5LbOA44GRuVpfC6fCCzPN3j7MXBGbmsQ\nMBnYHRgDTC4mNTMz615lztncLOkLtaOJsiJ5OT/sk6cADgAuzuUXAwfm+QOAyyPijYiYD8wDxkja\nCugfEXdFRJBud1CsU2vrSmBsjnNfYEZELIuI5cAMVicoMzPrZmXO2RwLfB1YJek1QKRc0r9RxXxk\n8gdgJHBORMyUNCQinsmrPAsMyfPbAHcVqj+Vy97M8/XltToLSQGtlPQisHmxvI06xfiOAY4BGD58\neKPN6bQRk65ruM6C0/fr8uc1M2s2DY9sIqJfRPSKiD4R0T8/bphoct1VEbELMJR0lLJT3fIgHe30\niIi4ICJGR8TowYMH91QYZmYtr0w3GpL2l/SjPH22s08SES8At5K6sp7LXWPkv4vzaouAYYVqQ3PZ\nojxfX75GHUm9gQHA0g7aMjOzHtAw2Ug6HTgJeChPJ0n6YYl6gyVtluc3Aj4FPAJMZ/XlbiYAV+f5\n6cCheYTZdqSBAHfnLrcVkvbI52MOr6tTa+sg4JZ8tHQDME7SwDwwYFwuMzOzHlDmnM1ngF3yyDQk\nXQzcC5zSoN5WwMX5vE0vYFpEXCvpf4FpkiYCT5BuW0BEzJU0jZTQVgInRMSq3NbxwEXARsD1eYJ0\na+pLJc0DlpFGsxERyySdBszK650aEctKbKuZmVWgTLIB2Iz0YQ6pq6qhiHgA2LWN8qXA2HbqTAGm\ntFE+G9ipjfLXgYPbaWsq6V48ZmbWw8okmx8C90q6lTQS7ePk38aYmZmV0TDZRMQvJd0G/Dlp5Ni3\nIuLZqgMzM7PWUbYb7aPAXqRk0xv4TWURmZlZyykzGu1c0g3U5gAPAsdKOqfqwMzMrHWUObL5JPBn\neUhxbTTa3EqjMjOzllLmR53zgOK1XIblMjMzs1LaPbKRdA3pHE0/4GFJd+fHuwN3d094ZmbWCjrq\nRvtRt0VhZmYtrd1kExG3d2cgZmbWukpdiNPMzGxdONmYmVnl2k02km7Of8/ovnDMzKwVdTRAYCtJ\nHwP2l3Q56bpob4uIeyqNzMzMWkZHyeafgH8k3XjsrLplQfqxp5mZWUMdjUa7ErhS0j9GxGndGJOZ\nmbWYMld9Pk3S/qRbCwDcFhHXVhuWmZm1kjIX4vwh77wt9A+qDszMzFpHmQtx7kfbt4X+dpWBmZlZ\n6yj7O5vNCvOlbgttZmZW49tCm5lZ5Tp7W2jwbaHNzKyTSt0WOiKeAaZXHIuZmbUoXxvNzMwq52Rj\nZmaV6zDZSNpA0iPdFYyZmbWmDpNNRKwCHpU0vJviMTOzFlRmgMBAYK6ku4FXaoURsX9lUZmZWUsp\nk2z+sfIozMyspZX5nc3tkrYFRkXETZI2BjaoPjQzM2sVZS7EeTRwJXB+LtoGuKrKoMzMrLWUGfp8\nArAnsAIgIh4DtqwyKDMzay1lks0bEfGn2gNJvUl36jQzMyulTLK5XdK3gY0kfQr4FXBNo0qShkm6\nVdJDkuZKOimXD5I0Q9Jj+e/AQp1TJM2T9KikfQvlu0mak5edLUm5vK+kK3L5TEkjCnUm5Od4TNKE\nsjvEzMy6XplkMwlYAswBjgV+C/xDiXorgZMjYgdgD+AESTvk9m6OiFHAzfkxedmhwI7AeOBcSbWB\nCOcBRwOj8jQ+l08ElkfESODHwBm5rUHAZGB3YAwwuZjUzMysezVMNvmmaRcDpwHfAy6OiIbdaBHx\nTETck+dfAh4mDS44ILdH/ntgnj8AuDwi3oiI+cA8YIykrYD+EXFXft5L6urU2roSGJuPevYFZkTE\nsohYDsxgdYIyM7Nu1nDos6T9gH8H/ki6n812ko6NiOvLPknu3toVmAkMyVeRBngWGJLntwHuKlR7\nKpe9mefry2t1FgJExEpJLwKbF8vbqGNmZt2szI86zwT2iYh5AJK2B64DSiUbSZsCvwa+FhEr8ukW\nACIiJPXYYANJxwDHAAwf7ivymJlVpcw5m5dqiSZ7HHipTOOS+pASzS8i4r9y8XO5a4z8d3EuXwQM\nK1QfmssW5fn68jXq5FFyA4ClHbS1hoi4ICJGR8TowYMHl9kkMzNbC+0mG0mfl/R5YLak30o6Io/q\nugaY1ajhfO7kQuDhiDirsGg6UBsdNgG4ulB+aB5hth1pIMDductthaQ9cpuH19WptXUQcEs+r3MD\nME7SwDwwYFwuMzOzHtBRN9rnCvPPAZ/I80uAjUq0vSfwVWCOpPty2beB04FpkiYCTwCHAETEXEnT\ngIdII9lOyFedBjgeuCg/7/Ws7sK7ELhU0jxgGWk0GxGxTNJprE6Kp0bEshIxm5lZBdpNNhFx5Lo0\nHBF3kAYUtGVsO3WmAFPaKJ8N7NRG+evAwe20NRWYWjZeMzOrTpnRaNsBfwuMKK7vWwyYmVlZZUaj\nXUXqrroGeKvacMzMrBWVSTavR8TZlUdiZmYtq0yy+VdJk4EbgTdqhbWrA5iZmTVSJtl8iDSq7JOs\n7kaL/NjMzKyhMsnmYOB9xdsMmJmZdUaZKwg8CGxWdSBmZta6yhzZbAY8ImkWa56z8dBnMzMrpUyy\nmVx5FGZm1tIaJpuIuL07AjEzs9ZV5goCL5FGnwFsCPQBXomI/lUGZmZmraPMkU2/2ny+6vIBpNs8\nm5mZlVJmNNrbIrmKdNtlMzOzUsp0o32+8LAXMBp4vbKIzMys5ZQZjVa8r81KYAGpK83MzKyUMuds\n1um+NmZmZu0mG0n/1EG9iIjTKojHzMxaUEdHNq+0UbYJMBHYHHCyMTOzUjq6LfSZtXlJ/YCTgCOB\ny4Ez26tnZmZWr8NzNpIGAV8HvgxcDHwkIpZ3R2BmZtY6Ojpn8y/A54ELgA9FxMvdFpWZmbWUjn7U\neTKwNfAPwNOSVuTpJUkruic8MzNrBR2ds+nU1QXMzMza44RiZmaVc7IxM7PKOdmYmVnlnGzMzKxy\nTjZmZlY5JxszM6uck42ZmVXOycbMzCrnZGNmZpWrLNlImippsaQHC2WDJM2Q9Fj+O7Cw7BRJ8yQ9\nKmnfQvlukubkZWdLUi7vK+mKXD5T0ohCnQn5OR6TNKGqbTQzs3KqPLK5CBhfVzYJuDkiRgE358dI\n2gE4FNgx1zlX0ga5znnA0cCoPNXanAgsj4iRwI+BM3Jbg4DJwO7AGGByMamZmVn3qyzZRMTvgGV1\nxQeQblVA/ntgofzyiHgjIuYD84AxkrYC+kfEXRERwCV1dWptXQmMzUc9+wIzImJZvh3CDN6Z9MzM\nrBt19zmbIRHxTJ5/FhiS57cBFhbWeyqXbZPn68vXqBMRK4EXSXcQba8tMzPrIT02QCAfqURPPT+A\npGMkzZY0e8mSJT0ZiplZS+vuZPNc7hoj/12cyxcBwwrrDc1li/J8ffkadST1BgYASzto6x0i4oKI\nGB0RowcPHrwOm2VmZh3p7mQzHaiNDpsAXF0oPzSPMNuONBDg7tzltkLSHvl8zOF1dWptHQTcko+W\nbgDGSRqYBwaMy2VmZtZD2r152rqS9Etgb2ALSU+RRoidDkyTNBF4AjgEICLmSpoGPASsBE6IiFW5\nqeNJI9s2Aq7PE8CFwKWS5pEGIhya21om6TRgVl7v1IioH6hgZmbdqLJkExGHtbNobDvrTwGmtFE+\nG9ipjfLXgYPbaWsqMLV0sGZmVilfQcDMzCrnZGNmZpVzsjEzs8o52ZiZWeWcbMzMrHJONmZmVjkn\nGzMzq5yTjZmZVc7JxszMKlfZFQSsa4yYdF2Hyxecvl83RWJmtvZ8ZGNmZpVzsjEzs8o52ZiZWeWc\nbMzMrHJONmZmVjknGzMzq5yTjZmZVc7JxszMKudkY2ZmlXOyMTOzyjnZmJlZ5ZxszMysck42ZmZW\nOScbMzOrnJONmZlVzsnGzMwq52RjZmaVc7IxM7PKOdmYmVnlnGzMzKxyTjZmZlY5JxszM6uck42Z\nmVWupZONpPGSHpU0T9Kkno7HzGx91bunA6iKpA2Ac4BPAU8BsyRNj4iHejay7jdi0nUdLl9w+n7d\n0oaZrb9aNtkAY4B5EfE4gKTLgQOA9S7ZNINGyQoaJywnPLN3L0VET8dQCUkHAeMj4qj8+KvA7hHx\nN4V1jgGOyQ8/ADzaoNktgOfXMbR1baMZYmiWNpohhq5ooxliaJY2miGGZmmjGWIo08a2ETG4USOt\nfGTTUERcAFxQdn1JsyNi9Lo857q20QwxNEsbzRBDV7TRDDE0SxvNEEOztNEMMXRVG9DaAwQWAcMK\nj4fmMjMz62atnGxmAaMkbSdpQ+BQYHoPx2Rmtl5q2W60iFgp6W+AG4ANgKkRMXcdmy3d5VZhG80Q\nQ7O00QwxdEUbzRBDs7TRDDE0SxvNEENXtdG6AwTMzKx5tHI3mpmZNQknGzMzq5yTjZmZVc7JphMk\n7SXpnJLrjpS0Zxvle0ravuujMzNrXi07Gq2rSNoV+BJwMDAf+K+SVX8CnNJG+Yq87HNrGc8WwNLo\n5pEdkoYA2+SHiyLiuXdjG10RQ7PE0QxtdNX+tNbnZNMGSe8HDsvT88AVpJF7+3SimSERMae+MCLm\nSBpRMo49gNOBZcBpwKWkS0f0knR4RPx32WDW9kNB0i7AvwMDWP2j2KGSXgCOj4h73g1tdEUMzRJH\nM7TRVfszt/VB0nUL335/AtMj4uHuqN9KbTRDDO2KCE91E/AWcDswslD2eCfbeKyDZfNKtjEbGEc6\nqloO7JHLPwjcW7KNXYC7gIeBm/L0SC77SIn695GuKVdfvgdwf8kYeryNroihWeJohja6cH9+K7c1\nCfhKnibVyqqu30ptNEMMHba9LpVbdQIOBC4HFgI/A8YC8zvZxi+Bo9soPwq4omQb9xXmH65bVjbZ\nrOuHSlckzR5voytiaJY4mqGNLtyf/wf0aaN8w46eo6vqt1IbzRBDR5O70doQEVcBV0nahHQ4+TVg\nS0nnAb+JiBtLNPM14DeSvgz8IZeNJr1of1kylLcK86/Vh1myjU0iYmZ9YUTclbevkeslXQdcQkq+\nkK45dzgNwAL/AAAHAElEQVRQthuvGdroihiaJY5maKOr9udbwNbAE3XlW7Hm+7+q+q3URjPE0C5f\nQaAkSQNJ3VlfjIixnai3D7BTfjg3Im7pRN1VwCuAgI2AV2uLgPdERJ8SbZwNbE/bHwrzo3DLhQ7a\n+DRt9+H+thPb0uNtdEUMzRJHM7TRRTGMB34KPMbq9+dwYCTwN9HgvOS61m+lNpohhg7bdrJpfV31\nIWtWBUm9SDc7LL4/Z0XEqu6o30ptNEMM7bbrZGNrS9Ixke4J9K5uoytiaJY4mqGNrtqf1lr8o871\nWL5T6To10RVhNEEbXRFDV7TTDPuiK9rokv0p6dqerN9KbTRFDD6yWX9JOjYizi+xXleN3d8GmBkR\nLxfKx5ftB5Y0BoiImCVpB2A88MjadgdKuiQiDl+buoU29iJ1OTxYZuCIpN1JIwtXSNqINKz0I8BD\nwA8i4sUSbZxIGqiysNG6HbRRu8fT0xFxk6QvAR8jDZG/ICLeLNHG+4DPk84BriKNZLosIlasbVx1\n7W8VEc/0VP1WaqMpYnCyWX9JOjIift5gnW+Rftx6OfBULh5K+qC6PCJOL/E8JwInkD7IdgFOioir\n87J7IuIjJdqYDHya9EPkGcDuwK3Ap4AbImJKg/r1N84TsA9wC0BE7N8ohtzO3RExJs8fnbfrN6Tf\nQ13TaH9ImgvsHOl+SxeQBn1cSRpev3NEfL5EDC+SBo78kTTE/lcRsaRM/IU2fkHalxsDLwCbkq6O\nMRYgIo5oUP9E4LPA74DPAPfmdv6S9KPO2zoTj7VN0pYRsbin4+gS6zJu2tO7ewKeLLFOV4zdnwNs\nmudHkH6selJ+XPb3QnNIN8HbmHTJn/65fCPggRL17wH+E9gb+ET++0ye/0Qn9tm9hflZwOA8vwkw\np0T9h4sx1S27r2wMpC7wccCFwBLScOMJQL+SbTyQ//YGngM2yI9Vcn/OKdTZGLgtzw8v+5rm9QeQ\nrpLxCOlKGUtJX0pOBzZbx/f39SXX6w/8kHSFji/VLTu3ZBvvBc4DzgE2B76b99E0YKuSbQyqmzYH\nFgADgUEl6o+v268XAg8Al5GuaFImhk2BU4G5wIv5vXUXcMS6vBYR4XM2rU7SA+1Mc4AhJZqojbuv\n15lx970id51FxALSB/2nJZ1F+f79lRGxKiJeBf4YuasmIl4rGcdo0u+dvgO8GOmb92sRcXtE3F4y\nBkiXChooaXPSh+2SHMcrwMoS9R+UdGSev1/SaHj7EkkNu66yiIi3IuLGiJhIen3OJXUrPt6J7dgQ\n6EdKFgNyeV+g4ZD6rPY7vb6kDyki4slO1If0Ybwc2DsiBkXE5qQjzuV5WYckfaSdaTfSUXQZPye9\nD38NHCrp15L65mV7lGzjIlJX6ELSEfdrpCO+/yFd1qeM50nv0do0m9T1fE+eb+QHhfkzSV+mPkf6\nUtSwuzz7Bek9tC/wPeBs4KvAPpJ+0FHFhtY1W3lq7on0rXUXYNu6aQSpv75R/fHAPOB60u1hLyB9\ni55H4ZtUgzZuAXapK+tN+u3PqpJtzAQ2zvO9CuUDqDtCaNDOUOBXpN8SNDyya6P+gvzPOD//3SqX\nb0qJI5Mc70WkLrCZpATzOOnySDuXjKHdI4faPirRxt/l530COBG4mXS1jDnA5BL1TyJ9a/4Z6ajk\nyFw+GPhdJ/bno2uzrLDOqvz+urWN6bWSMdxX9/g7wO9JRxal3lusecT7ZEftd9DGyfl/60OFsvmd\n2Jf3tPecnYjh/rrHs/LfXqTzo536f1mjrXWp7Kn5J9Kh9F7tLLusZBu9SN/wvpCnPchdKCXrDwXe\n286yPUu20bed8i2K/5ydiGk/0gn5rtrPGwPbdWL9/sDOwG6U7OIo1H1/F8W8NbB1nt8MOAgY04n6\nO+Y6H1yHGG4EvlncB6Qj7m8BN5Wo/yAwqp1lC0vG8DCFLzC57AhSV9ITJdu4vzD//bplDbtXC+vW\nvgydRTrqLH1NRtI51a/npDWffE4+L2vYNZrXu7P2eQHsTzofWlvWMPl3NHmAgJn1mHxljkmk0Y5b\n5uLngOnA6RGxvEH9g0gf5o+2sezASJeeahTDPwM3RsRNdeXjgX+LiFEl2jgV+OcojLTM5SPzdhzU\nqI26evsD3wZGRMR7S9aZXFd0bkQskfTeHFvDkZeSPgz8BzCKlGz/KiL+T9Jg4LCIOLsz27FG2042\nZtaMyoyWrLJ+T7eRh8ZvHxEPtsK+cLIxs6Yk6cmIGN5T9VupjWaIwVd9NrMeI+mB9hZRYrTkutZv\npTaaIYaOONmYWU8aQhpmW39uRqST1VXXb6U2miGGdjnZmFlPupb0g9/76hdIuq0b6rdSG80QQ7t8\nzsbMzCrnKwiYmVnlnGzMzKxyTjZmZlY5JxszM6uck42ZmVXu/wNCP2yz4fPUrgAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xa627b068d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# the variable is also discrete in nature. A person can have 1, 2 accounts but not 2.3 accounts\n",
    "# let's inspect the number of observations per value of the variable\n",
    "\n",
    "fig = data.open_il_24m.value_counts().plot.bar()\n",
    "fig.set_title('Number of installment accounts open')\n",
    "fig.set_ylabel('Number of borrowers')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "This is how a mixed variable looks like!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    C\n",
       "1    A\n",
       "2    A\n",
       "3    A\n",
       "4    A\n",
       "Name: open_il_24m, dtype: object"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.open_il_24m.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Engineering mixed types of variables\n",
    "\n",
    "Below I will demonstrate how to engineer mixed types of variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>open_il_24m</th>\n",
       "      <th>open_il_24m_numerical</th>\n",
       "      <th>open_il_24m_categorical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1077501</td>\n",
       "      <td>C</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1077430</td>\n",
       "      <td>A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1077175</td>\n",
       "      <td>A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1076863</td>\n",
       "      <td>A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1075358</td>\n",
       "      <td>A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id open_il_24m open_il_24m_numerical open_il_24m_categorical\n",
       "0  1077501           C                   NaN                       C\n",
       "1  1077430           A                   NaN                       A\n",
       "2  1077175           A                   NaN                       A\n",
       "3  1076863           A                   NaN                       A\n",
       "4  1075358           A                   NaN                       A"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we create 2 variables, a numerical one containing the numerical part, and\n",
    "# a categorical variable with the codes (strings)\n",
    "\n",
    "data['open_il_24m_numerical'] = np.where(data.open_il_24m.str.isdigit(), data.open_il_24m, np.nan)\n",
    "data['open_il_24m_categorical'] = np.where(data.open_il_24m.str.isdigit(), np.nan, data.open_il_24m,)\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The categorical variable captures the categorical part of the mixed original variable. Where the categorical variable contains a value, the numerical variable contains NA, and vice versa. See below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>open_il_24m</th>\n",
       "      <th>open_il_24m_numerical</th>\n",
       "      <th>open_il_24m_categorical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>466285</th>\n",
       "      <td>68444620</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466286</th>\n",
       "      <td>68547583</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466287</th>\n",
       "      <td>67849662</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466288</th>\n",
       "      <td>68506885</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466289</th>\n",
       "      <td>68341763</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466290</th>\n",
       "      <td>68416935</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466291</th>\n",
       "      <td>68466916</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466292</th>\n",
       "      <td>68407277</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466293</th>\n",
       "      <td>68426831</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466294</th>\n",
       "      <td>68377020</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466295</th>\n",
       "      <td>68366663</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466296</th>\n",
       "      <td>68476807</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466297</th>\n",
       "      <td>68585839</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466298</th>\n",
       "      <td>68466922</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466299</th>\n",
       "      <td>68506798</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466300</th>\n",
       "      <td>68495092</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466301</th>\n",
       "      <td>68476734</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466302</th>\n",
       "      <td>66310712</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466303</th>\n",
       "      <td>68616873</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466304</th>\n",
       "      <td>68466961</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466305</th>\n",
       "      <td>68617057</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466306</th>\n",
       "      <td>68387134</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466307</th>\n",
       "      <td>68466995</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466308</th>\n",
       "      <td>68396080</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466309</th>\n",
       "      <td>68516507</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466310</th>\n",
       "      <td>68009401</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466311</th>\n",
       "      <td>68354783</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466312</th>\n",
       "      <td>68415473</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466313</th>\n",
       "      <td>68446820</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>466314</th>\n",
       "      <td>68596180</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488789</th>\n",
       "      <td>66546955</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488794</th>\n",
       "      <td>66575817</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488805</th>\n",
       "      <td>66526187</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488816</th>\n",
       "      <td>66446348</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488825</th>\n",
       "      <td>66595970</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488854</th>\n",
       "      <td>66645795</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488905</th>\n",
       "      <td>66836112</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488917</th>\n",
       "      <td>66786120</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488924</th>\n",
       "      <td>66575722</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488928</th>\n",
       "      <td>66876118</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488955</th>\n",
       "      <td>66505743</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488961</th>\n",
       "      <td>66756118</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488962</th>\n",
       "      <td>66726112</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488976</th>\n",
       "      <td>66806116</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488998</th>\n",
       "      <td>66826113</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489014</th>\n",
       "      <td>66606538</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489016</th>\n",
       "      <td>66756123</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489024</th>\n",
       "      <td>66415507</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489026</th>\n",
       "      <td>66686122</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489042</th>\n",
       "      <td>66746120</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489059</th>\n",
       "      <td>66756112</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489061</th>\n",
       "      <td>66796115</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489068</th>\n",
       "      <td>66546772</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489089</th>\n",
       "      <td>66926112</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489092</th>\n",
       "      <td>66446164</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489099</th>\n",
       "      <td>66686112</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489134</th>\n",
       "      <td>66866115</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489145</th>\n",
       "      <td>66746115</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489160</th>\n",
       "      <td>66636338</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489161</th>\n",
       "      <td>66675113</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>21372 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              id open_il_24m open_il_24m_numerical open_il_24m_categorical\n",
       "466285  68444620           0                     0                     NaN\n",
       "466286  68547583           1                     1                     NaN\n",
       "466287  67849662           0                     0                     NaN\n",
       "466288  68506885           2                     2                     NaN\n",
       "466289  68341763           4                     4                     NaN\n",
       "466290  68416935           2                     2                     NaN\n",
       "466291  68466916           1                     1                     NaN\n",
       "466292  68407277           1                     1                     NaN\n",
       "466293  68426831           0                     0                     NaN\n",
       "466294  68377020           0                     0                     NaN\n",
       "466295  68366663           0                     0                     NaN\n",
       "466296  68476807           3                     3                     NaN\n",
       "466297  68585839           1                     1                     NaN\n",
       "466298  68466922           1                     1                     NaN\n",
       "466299  68506798           0                     0                     NaN\n",
       "466300  68495092           1                     1                     NaN\n",
       "466301  68476734           0                     0                     NaN\n",
       "466302  66310712           1                     1                     NaN\n",
       "466303  68616873           3                     3                     NaN\n",
       "466304  68466961           0                     0                     NaN\n",
       "466305  68617057           1                     1                     NaN\n",
       "466306  68387134           4                     4                     NaN\n",
       "466307  68466995           0                     0                     NaN\n",
       "466308  68396080           2                     2                     NaN\n",
       "466309  68516507           2                     2                     NaN\n",
       "466310  68009401           2                     2                     NaN\n",
       "466311  68354783           0                     0                     NaN\n",
       "466312  68415473           4                     4                     NaN\n",
       "466313  68446820           0                     0                     NaN\n",
       "466314  68596180           1                     1                     NaN\n",
       "...          ...         ...                   ...                     ...\n",
       "488789  66546955           2                     2                     NaN\n",
       "488794  66575817           2                     2                     NaN\n",
       "488805  66526187           0                     0                     NaN\n",
       "488816  66446348           1                     1                     NaN\n",
       "488825  66595970           2                     2                     NaN\n",
       "488854  66645795           1                     1                     NaN\n",
       "488905  66836112           6                     6                     NaN\n",
       "488917  66786120           2                     2                     NaN\n",
       "488924  66575722           4                     4                     NaN\n",
       "488928  66876118           1                     1                     NaN\n",
       "488955  66505743           2                     2                     NaN\n",
       "488961  66756118           2                     2                     NaN\n",
       "488962  66726112           0                     0                     NaN\n",
       "488976  66806116           4                     4                     NaN\n",
       "488998  66826113           1                     1                     NaN\n",
       "489014  66606538           2                     2                     NaN\n",
       "489016  66756123          11                    11                     NaN\n",
       "489024  66415507           1                     1                     NaN\n",
       "489026  66686122           0                     0                     NaN\n",
       "489042  66746120           2                     2                     NaN\n",
       "489059  66756112           0                     0                     NaN\n",
       "489061  66796115           0                     0                     NaN\n",
       "489068  66546772           3                     3                     NaN\n",
       "489089  66926112           3                     3                     NaN\n",
       "489092  66446164           0                     0                     NaN\n",
       "489099  66686112           1                     1                     NaN\n",
       "489134  66866115           2                     2                     NaN\n",
       "489145  66746115           3                     3                     NaN\n",
       "489160  66636338           0                     0                     NaN\n",
       "489161  66675113           2                     2                     NaN\n",
       "\n",
       "[21372 rows x 4 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's inspect those instances of the dataset where numerical is not null\n",
    "# we can see that when the numerical variable is not null the categorical is null\n",
    "# and vice versa\n",
    "\n",
    "data.dropna(subset = ['open_il_24m_numerical'], axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we are ready to use those variables in machine learning. We should pre-process them as we learnt in previous sections. For example,  we could fill the missing values of the categorical part with the new category 'Missing'. And then we could fill the NA of the numerical  part with a value far out in the distribution.\n",
    "\n",
    "### Other types of mixed variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Cabin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>C85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>113803</td>\n",
       "      <td>C123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Survived            Ticket Cabin\n",
       "0         0         A/5 21171   NaN\n",
       "1         1          PC 17599   C85\n",
       "2         1  STON/O2. 3101282   NaN\n",
       "3         1            113803  C123\n",
       "4         0            373450   NaN"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load again the titanic dataset for demonstration\n",
    "\n",
    "data = pd.read_csv('titanic.csv', usecols = ['Ticket', 'Cabin', 'Survived'])\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variables Ticket and Cabin, are not strictly mixed in the sense that they do not contain numbers OR strings on different observations. They contains both numbers and strings in the same observation. Perhaps, if we separate them, we can add some information to the dataset. \n",
    "\n",
    "Let's have a look at what we could do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Sole\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:3: FutureWarning: currently extract(expand=None) means expand=False (return Index/Series/DataFrame) but in a future version of pandas this will be changed to expand=True (return DataFrame)\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Cabin_numerical</th>\n",
       "      <th>Cabin_categorical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C85</td>\n",
       "      <td>85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C123</td>\n",
       "      <td>123</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Cabin Cabin_numerical Cabin_categorical\n",
       "0   NaN             NaN               NaN\n",
       "1   C85              85                 C\n",
       "2   NaN             NaN               NaN\n",
       "3  C123             123                 C\n",
       "4   NaN             NaN               NaN"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for Cabin, it is relatively straightforward, we can extract the letters and the numbers in different variables\n",
    "\n",
    "data['Cabin_numerical'] = data.Cabin.str.extract('(\\d+)') # captures numerical part\n",
    "data['Cabin_categorical'] = data['Cabin'].str[0] # captures the first letter\n",
    "\n",
    "data[['Cabin', 'Cabin_numerical', 'Cabin_categorical']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['A/5 21171', 'PC 17599', 'STON/O2. 3101282', '113803', '373450',\n",
       "       '330877', '17463', '349909', '347742', '237736', 'PP 9549',\n",
       "       '113783', 'A/5. 2151', '347082', '350406', '248706', '382652',\n",
       "       '244373', '345763', '2649', '239865', '248698', '330923', '113788',\n",
       "       '347077', '2631', '19950', '330959', '349216', 'PC 17601',\n",
       "       'PC 17569', '335677', 'C.A. 24579', 'PC 17604', '113789', '2677',\n",
       "       'A./5. 2152', '345764', '2651', '7546', '11668', '349253',\n",
       "       'SC/Paris 2123', '330958', 'S.C./A.4. 23567', '370371', '14311',\n",
       "       '2662', '349237', '3101295', 'A/4. 39886', 'PC 17572', '2926',\n",
       "       '113509', '19947', 'C.A. 31026', '2697', 'C.A. 34651', 'CA 2144',\n",
       "       '2669', '113572', '36973', '347088', 'PC 17605', '2661',\n",
       "       'C.A. 29395', 'S.P. 3464', '3101281', '315151', 'C.A. 33111',\n",
       "       'S.O.C. 14879', '2680', '1601', '348123', '349208', '374746',\n",
       "       '248738', '364516', '345767', '345779', '330932', '113059',\n",
       "       'SO/C 14885', '3101278', 'W./C. 6608', 'SOTON/OQ 392086', '343275',\n",
       "       '343276', '347466', 'W.E.P. 5734', 'C.A. 2315', '364500', '374910',\n",
       "       'PC 17754', 'PC 17759', '231919', '244367', '349245', '349215',\n",
       "       '35281', '7540', '3101276', '349207', '343120', '312991', '349249',\n",
       "       '371110', '110465', '2665', '324669', '4136', '2627',\n",
       "       'STON/O 2. 3101294', '370369', 'PC 17558', 'A4. 54510', '27267',\n",
       "       '370372', 'C 17369', '2668', '347061', '349241',\n",
       "       'SOTON/O.Q. 3101307', 'A/5. 3337', '228414', 'C.A. 29178',\n",
       "       'SC/PARIS 2133', '11752', '7534', 'PC 17593', '2678', '347081',\n",
       "       'STON/O2. 3101279', '365222', '231945', 'C.A. 33112', '350043',\n",
       "       '230080', '244310', 'S.O.P. 1166', '113776', 'A.5. 11206',\n",
       "       'A/5. 851', 'Fa 265302', 'PC 17597', '35851', 'SOTON/OQ 392090',\n",
       "       '315037', 'CA. 2343', '371362', 'C.A. 33595', '347068', '315093',\n",
       "       '363291', '113505', 'PC 17318', '111240', 'STON/O 2. 3101280',\n",
       "       '17764', '350404', '4133', 'PC 17595', '250653', 'LINE',\n",
       "       'SC/PARIS 2131', '230136', '315153', '113767', '370365', '111428',\n",
       "       '364849', '349247', '234604', '28424', '350046', 'PC 17610',\n",
       "       '368703', '4579', '370370', '248747', '345770', '3101264', '2628',\n",
       "       'A/5 3540', '347054', '2699', '367231', '112277',\n",
       "       'SOTON/O.Q. 3101311', 'F.C.C. 13528', 'A/5 21174', '250646',\n",
       "       '367229', '35273', 'STON/O2. 3101283', '243847', '11813',\n",
       "       'W/C 14208', 'SOTON/OQ 392089', '220367', '21440', '349234',\n",
       "       '19943', 'PP 4348', 'SW/PP 751', 'A/5 21173', '236171', '347067',\n",
       "       '237442', 'C.A. 29566', 'W./C. 6609', '26707', 'C.A. 31921',\n",
       "       '28665', 'SCO/W 1585', '367230', 'W./C. 14263', 'STON/O 2. 3101275',\n",
       "       '2694', '19928', '347071', '250649', '11751', '244252', '362316',\n",
       "       '113514', 'A/5. 3336', '370129', '2650', 'PC 17585', '110152',\n",
       "       'PC 17755', '230433', '384461', '110413', '112059', '382649',\n",
       "       'C.A. 17248', '347083', 'PC 17582', 'PC 17760', '113798', '250644',\n",
       "       'PC 17596', '370375', '13502', '347073', '239853', 'C.A. 2673',\n",
       "       '336439', '347464', '345778', 'A/5. 10482', '113056', '349239',\n",
       "       '345774', '349206', '237798', '370373', '19877', '11967',\n",
       "       'SC/Paris 2163', '349236', '349233', 'PC 17612', '2693', '113781',\n",
       "       '19988', '9234', '367226', '226593', 'A/5 2466', '17421',\n",
       "       'PC 17758', 'P/PP 3381', 'PC 17485', '11767', 'PC 17608', '250651',\n",
       "       '349243', 'F.C.C. 13529', '347470', '29011', '36928', '16966',\n",
       "       'A/5 21172', '349219', '234818', '345364', '28551', '111361',\n",
       "       '113043', 'PC 17611', '349225', '7598', '113784', '248740',\n",
       "       '244361', '229236', '248733', '31418', '386525', 'C.A. 37671',\n",
       "       '315088', '7267', '113510', '2695', '2647', '345783', '237671',\n",
       "       '330931', '330980', 'SC/PARIS 2167', '2691', 'SOTON/O.Q. 3101310',\n",
       "       'C 7076', '110813', '2626', '14313', 'PC 17477', '11765', '3101267',\n",
       "       '323951', 'C 7077', '113503', '2648', '347069', 'PC 17757', '2653',\n",
       "       'STON/O 2. 3101293', '349227', '27849', '367655', 'SC 1748',\n",
       "       '113760', '350034', '3101277', '350052', '350407', '28403',\n",
       "       '244278', '240929', 'STON/O 2. 3101289', '341826', '4137', '315096',\n",
       "       '28664', '347064', '29106', '312992', '349222', '394140',\n",
       "       'STON/O 2. 3101269', '343095', '28220', '250652', '28228', '345773',\n",
       "       '349254', 'A/5. 13032', '315082', '347080', 'A/4. 34244', '2003',\n",
       "       '250655', '364851', 'SOTON/O.Q. 392078', '110564', '376564',\n",
       "       'SC/AH 3085', 'STON/O 2. 3101274', '13507', 'C.A. 18723', '345769',\n",
       "       '347076', '230434', '65306', '33638', '113794', '2666', '113786',\n",
       "       '65303', '113051', '17453', 'A/5 2817', '349240', '13509', '17464',\n",
       "       'F.C.C. 13531', '371060', '19952', '364506', '111320', '234360',\n",
       "       'A/S 2816', 'SOTON/O.Q. 3101306', '113792', '36209', '323592',\n",
       "       '315089', 'SC/AH Basle 541', '7553', '31027', '3460', '350060',\n",
       "       '3101298', '239854', 'A/5 3594', '4134', '11771', 'A.5. 18509',\n",
       "       '65304', 'SOTON/OQ 3101317', '113787', 'PC 17609', 'A/4 45380',\n",
       "       '36947', 'C.A. 6212', '350035', '315086', '364846', '330909',\n",
       "       '4135', '26360', '111427', 'C 4001', '382651', 'SOTON/OQ 3101316',\n",
       "       'PC 17473', 'PC 17603', '349209', '36967', 'C.A. 34260', '226875',\n",
       "       '349242', '12749', '349252', '2624', '2700', '367232',\n",
       "       'W./C. 14258', 'PC 17483', '3101296', '29104', '2641', '2690',\n",
       "       '315084', '113050', 'PC 17761', '364498', '13568', 'WE/P 5735',\n",
       "       '2908', '693', 'SC/PARIS 2146', '244358', '330979', '2620',\n",
       "       '347085', '113807', '11755', '345572', '372622', '349251', '218629',\n",
       "       'SOTON/OQ 392082', 'SOTON/O.Q. 392087', 'A/4 48871', '349205',\n",
       "       '2686', '350417', 'S.W./PP 752', '11769', 'PC 17474', '14312',\n",
       "       'A/4. 20589', '358585', '243880', '2689', 'STON/O 2. 3101286',\n",
       "       '237789', '13049', '3411', '237565', '13567', '14973', 'A./5. 3235',\n",
       "       'STON/O 2. 3101273', 'A/5 3902', '364848', 'SC/AH 29037', '248727',\n",
       "       '2664', '349214', '113796', '364511', '111426', '349910', '349246',\n",
       "       '113804', 'SOTON/O.Q. 3101305', '370377', '364512', '220845',\n",
       "       '31028', '2659', '11753', '350029', '54636', '36963', '219533',\n",
       "       '349224', '334912', '27042', '347743', '13214', '112052', '237668',\n",
       "       'STON/O 2. 3101292', '350050', '349231', '13213', 'S.O./P.P. 751',\n",
       "       'CA. 2314', '349221', '8475', '330919', '365226', '349223', '29751',\n",
       "       '2623', '5727', '349210', 'STON/O 2. 3101285', '234686', '312993',\n",
       "       'A/5 3536', '19996', '29750', 'F.C. 12750', 'C.A. 24580', '244270',\n",
       "       '239856', '349912', '342826', '4138', '330935', '6563', '349228',\n",
       "       '350036', '24160', '17474', '349256', '2672', '113800', '248731',\n",
       "       '363592', '35852', '348121', 'PC 17475', '36864', '350025',\n",
       "       '223596', 'PC 17476', 'PC 17482', '113028', '7545', '250647',\n",
       "       '348124', '34218', '36568', '347062', '350048', '12233', '250643',\n",
       "       '113806', '315094', '36866', '236853', 'STON/O2. 3101271', '239855',\n",
       "       '28425', '233639', '349201', '349218', '16988', '376566',\n",
       "       'STON/O 2. 3101288', '250648', '113773', '335097', '29103',\n",
       "       '392096', '345780', '349204', '350042', '29108', '363294',\n",
       "       'SOTON/O2 3101272', '2663', '347074', '112379', '364850', '8471',\n",
       "       '345781', '350047', 'S.O./P.P. 3', '2674', '29105', '347078',\n",
       "       '383121', '36865', '2687', '113501', 'W./C. 6607',\n",
       "       'SOTON/O.Q. 3101312', '374887', '3101265', '12460', 'PC 17600',\n",
       "       '349203', '28213', '17465', '349244', '2685', '2625', '347089',\n",
       "       '347063', '112050', '347087', '248723', '3474', '28206', '364499',\n",
       "       '112058', 'STON/O2. 3101290', 'S.C./PARIS 2079', 'C 7075', '315098',\n",
       "       '19972', '368323', '367228', '2671', '347468', '2223', 'PC 17756',\n",
       "       '315097', '392092', '11774', 'SOTON/O2 3101287', '2683', '315090',\n",
       "       'C.A. 5547', '349213', '347060', 'PC 17592', '392091', '113055',\n",
       "       '2629', '350026', '28134', '17466', '233866', '236852',\n",
       "       'SC/PARIS 2149', 'PC 17590', '345777', '349248', '695', '345765',\n",
       "       '2667', '349212', '349217', '349257', '7552', 'C.A./SOTON 34068',\n",
       "       'SOTON/OQ 392076', '211536', '112053', '111369', '370376'], dtype=object)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ticket is not as clear...but we could still capture the first part of the ticket as a code (category)\n",
    "# and the second part of the ticket as numeric\n",
    "\n",
    "data.Ticket.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Ticket_numerical</th>\n",
       "      <th>Ticket_categorical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>21171</td>\n",
       "      <td>A/5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>PC 17599</td>\n",
       "      <td>17599</td>\n",
       "      <td>PC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>3101282</td>\n",
       "      <td>STON/O2.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>113803</td>\n",
       "      <td>113803</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>373450</td>\n",
       "      <td>373450</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>330877</td>\n",
       "      <td>330877</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>17463</td>\n",
       "      <td>17463</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>349909</td>\n",
       "      <td>349909</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>347742</td>\n",
       "      <td>347742</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>237736</td>\n",
       "      <td>237736</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Ticket Ticket_numerical Ticket_categorical\n",
       "0         A/5 21171            21171                A/5\n",
       "1          PC 17599            17599                 PC\n",
       "2  STON/O2. 3101282          3101282           STON/O2.\n",
       "3            113803           113803                NaN\n",
       "4            373450           373450                NaN\n",
       "5            330877           330877                NaN\n",
       "6             17463            17463                NaN\n",
       "7            349909           349909                NaN\n",
       "8            347742           347742                NaN\n",
       "9            237736           237736                NaN"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# extract the last bit of ticket as number\n",
    "data['Ticket_numerical'] = data.Ticket.apply(lambda s: s.split()[-1])\n",
    "data['Ticket_numerical'] = np.where(data.Ticket_numerical.str.isdigit(), data.Ticket_numerical, np.nan)\n",
    "\n",
    "# extract the first part of ticket as category\n",
    "data['Ticket_categorical'] = data.Ticket.apply(lambda s: s.split()[0])\n",
    "data['Ticket_categorical'] = np.where(data.Ticket_categorical.str.isdigit(), np.nan, data.Ticket_categorical)\n",
    "\n",
    "data[['Ticket', 'Ticket_numerical','Ticket_categorical']].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ticket_original no of labels:  681\n",
      "Cabin_original no of labels:  148\n",
      "Ticket_categorical no of labels:  44\n",
      "Cabin_categorical no of labels:  9\n"
     ]
    }
   ],
   "source": [
    "# let's compare the number of categories of the newly designed variables\n",
    "\n",
    "print('Ticket_original no of labels: ', len(data.Ticket.unique()))\n",
    "print('Cabin_original no of labels: ', len(data.Cabin.unique()))\n",
    "\n",
    "print('Ticket_categorical no of labels: ', len(data.Ticket_categorical.unique()))\n",
    "print('Cabin_categorical no of labels: ', len(data.Cabin_categorical.unique()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see how we reduced quite a bit the number of categories with this simple pre-processing of the variables. Whether these newly designed variables are or not predictive of survival, remains to be tested. I will come back to this in the last section of this course. Meanwhile, go ahead and built machine learning algorithms with these new variables, and see if they boost the performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
